ESXi Host experiences All Paths Down (APD) and VM crashes during Nutanix AOS upgrades.

Products

• VMware vSphere ESXi

Issue/Introduction

During a Nutanix cluster upgrade or maintenance event involving a Controller VM (CVM) reboot, the ESXi host loses connectivity to the Nutanix NFS datastore.

The datastore enters an extended All Paths Down (APD) state. Once the standard 140-second APD timeout expires, ESXi begins fast-failing I/O. As a result, guest virtual machines (frequently Linux VMs) experience fatal SCSI resets, causing their operating systems to freeze, crash, or enter a read-only state.

/var/run/log/vobd.log
The APD timer will start and eventually expire, fast-failing guest I/O:

[vob.storage.apd.start] Device or filesystem with identifier [Datastore_ID] has entered the All Paths Down state.

[vob.storage.apd.timeout] Device or filesystem with identifier [Datastore_ID] has entered the All Paths Down Timeout state after being in the All Paths Down state for 140 seconds. I/Os will now be fast failed.

/var/run/log/vmkernel.log
The host will log continuous NFS connection failures and SCSI aborts from the struggling guest VMs:

NFS: 7038: Status:No connection. Retrying synchronous write I/O 25 of 25 times

PVSCSI: 2769: scsi0:0: SCSI ABORT ctx=...

Additional symptoms reported

Linux guest operating systems suddenly enter a read-only file system state or become completely unresponsive during the upgrade window.

vCenter Server and VMware Aria Operations fail to generate or display active APD alerts, despite the storage disconnection lasting for an extended period (e.g., over 60 minutes).

ESXi host performance metrics show low-to-normal CPU and Memory utilization during the APD event, ruling out resource exhaustion.

 

Environment

VMware vSphere ESXi 7.0.x / 8.0.x

Nutanix AOS versions 7.5.1.1 and 7.5.1.2

Server hardware utilizing a virtual USB management interface (e.g., Dell iDRAC, HPE iLO)

Cause

This issue is caused by a known Nutanix AOS defect, documented internally by Nutanix under Engineering ID ENG-921260.

When a local CVM reboots during an upgrade, the ESXi host relies on a static route to temporarily redirect storage traffic to a remote CVM on another host. However, a bug in the AOS networking script incorrectly identifies the ESXi host's virtual USB management interface (vusb0 or vSwitchBMC) as a legitimate network uplink.

Because the script attempts to route the static storage traffic through the virtual USB interface instead of the physical network adapters (vmnics), the traffic is dropped. The ESXi host cannot reach the remote CVM, resulting in a persistent APD state that lasts until the local CVM finishes booting and resumes normal storage operations (often taking 60 to 80 minutes).

Note: Host performance metrics during the event will typically show normal/healthy CPU and Memory utilization, confirming that the APD is not caused by host-level resource exhaustion or a DRS vMotion storm.

Resolution

To resolve this issue, upgrade the Nutanix cluster to AOS version 7.5.1.6 or later.

The patched AOS release contains the fix for ENG-921260, which updates the networking script to properly ignore the vusb0 interface and correctly route storage traffic during CVM maintenance reboots.

Workaround: If the environment is currently running an affected version (7.5.1.1 or 7.5.1.2), halt all further rolling upgrades or host reboots until the fixed AOS version can be applied.

Additional Information

Related Knowledge Base Article	When This KB Applies
https://knowledge.broadcom.com/external/article/376173/esxi-creates-standard-switch-vswitchbmc.html	Applies when ESXi 8.0 U3 automatically generates the vSwitchBMC standard switch and vusb0 uplink for out-of-band management.
KB 426660: Alarm: "Lost network connectivity on virtual switch vSwitchBMC. Physical NIC vusb0 is down."	Applies when vCenter triggers a false-positive network connectivity alarm indicating that the vusb0 physical NIC is down.
KB 383014: ESXi host unexpectedly creates vmkernel ports from portgroups on the vSwitch	Applies when VMkernel ports are unexpectedly created on the vSwitch, often associated with the BMC virtual USB network adapter initialization.
KB 338061: ESXi hosts in a Nutanix cluster experience APDs	Applies when Nutanix NFS datastores experience APD events due to ProductLockerLocation (VMware Tools) misconfiguration during a CVM reboot.